Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61023

Jenkins agent threads deadlock due to remoting.jar

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      ref) https://wiki.jenkins.io/display/JENKINS/Remoting+issue

       

      description:

      we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because SynchronousCommandTransport$ReaderThread.run from remoting.jar. (ref: https://wiki.jenkins.io/display/JENKINS/Remoting+issue)

      I added some screenshots. I really appreciate if you take a look and help us to find root cause.

       

      agent server stats:

       

      cpu usage:

       

      tracking PID 15699: ( It was remoting.jar )

       

      agentthread dump:  

       

       

      Thanks!

        Attachments

        1. channel thread.png
          channel thread.png
          77 kB
        2. cpu usage of remoting_jar.png
          cpu usage of remoting_jar.png
          20 kB
        3. remoting_jar ps.png
          remoting_jar ps.png
          13 kB
        4. stats.png
          stats.png
          134 kB

          Activity

          doubleknd26 Kideok Kim created issue -
          doubleknd26 Kideok Kim made changes -
          Field Original Value New Value
          Attachment channel thread.png [ 50164 ]
          Attachment cpu usage of remoting_jar.png [ 50165 ]
          Attachment remoting_jar ps.png [ 50166 ]
          Attachment stats.png [ 50167 ]
          doubleknd26 Kideok Kim made changes -
          Description *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !image-2020-02-09-23-46-42-173.png|width=925,height=312!

           

          *cpu usage:*

          *!image-2020-02-09-23-48-03-663.png!*

           

          *tracking PID 15699: (* It was remoting.jar )

          *!image-2020-02-09-23-48-19-132.png!*

           

          *slave thread dump:*  !image-2020-02-09-23-47-26-493.png|width=849,height=291!

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          doubleknd26 Kideok Kim made changes -
          Description *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          doubleknd26 Kideok Kim made changes -
          Description *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!
          Hide
          jthompson Jeff Thompson added a comment -

          As described on the page you link to, you'll need to provide further information before anything can be done with this report. There is insufficient here for any action.

          Things you will need to investigate include: versions used, types of agents used, operations of the different threads, what the agents are doing, etc.

          Most of the time, issues like this are caused by something external to the Remoting library. These things can be system, network, or configuration. It might be a misbehaving plugin or interaction between multiple plugins.

          Issues like this usually cannot be addressed without providing a reproducible scenario or sufficient diagnostics. Best of luck in troubleshooting and isolating the problem.

          Show
          jthompson Jeff Thompson added a comment - As described on the page you link to, you'll need to provide further information before anything can be done with this report. There is insufficient here for any action. Things you will need to investigate include: versions used, types of agents used, operations of the different threads, what the agents are doing, etc. Most of the time, issues like this are caused by something external to the Remoting library. These things can be system, network, or configuration. It might be a misbehaving plugin or interaction between multiple plugins. Issues like this usually cannot be addressed without providing a reproducible scenario or sufficient diagnostics. Best of luck in troubleshooting and isolating the problem.
          slide_o_mix Alex Earl made changes -
          Summary Jenkins slave threads deadlock due to remoting.jar Jenkins agent threads deadlock due to remoting.jar
          slide_o_mix Alex Earl made changes -
          Description ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!
          ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *agent server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *agentthread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            doubleknd26 Kideok Kim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: