Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61023

Jenkins agent threads deadlock due to remoting.jar

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • core, remoting
    • None

      ref) https://wiki.jenkins.io/display/JENKINS/Remoting+issue

       

      description:

      we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because SynchronousCommandTransport$ReaderThread.run from remoting.jar. (ref: https://wiki.jenkins.io/display/JENKINS/Remoting+issue)

      I added some screenshots. I really appreciate if you take a look and help us to find root cause.

       

      agent server stats:

       

      cpu usage:

       

      tracking PID 15699: ( It was remoting.jar )

       

      agentthread dump:  

       

       

      Thanks!

        1. channel thread.png
          channel thread.png
          77 kB
        2. cpu usage of remoting_jar.png
          cpu usage of remoting_jar.png
          20 kB
        3. remoting_jar ps.png
          remoting_jar ps.png
          13 kB
        4. stats.png
          stats.png
          134 kB

          [JENKINS-61023] Jenkins agent threads deadlock due to remoting.jar

          Jeff Thompson added a comment -

          As described on the page you link to, you'll need to provide further information before anything can be done with this report. There is insufficient here for any action.

          Things you will need to investigate include: versions used, types of agents used, operations of the different threads, what the agents are doing, etc.

          Most of the time, issues like this are caused by something external to the Remoting library. These things can be system, network, or configuration. It might be a misbehaving plugin or interaction between multiple plugins.

          Issues like this usually cannot be addressed without providing a reproducible scenario or sufficient diagnostics. Best of luck in troubleshooting and isolating the problem.

          Jeff Thompson added a comment - As described on the page you link to, you'll need to provide further information before anything can be done with this report. There is insufficient here for any action. Things you will need to investigate include: versions used, types of agents used, operations of the different threads, what the agents are doing, etc. Most of the time, issues like this are caused by something external to the Remoting library. These things can be system, network, or configuration. It might be a misbehaving plugin or interaction between multiple plugins. Issues like this usually cannot be addressed without providing a reproducible scenario or sufficient diagnostics. Best of luck in troubleshooting and isolating the problem.

            Unassigned Unassigned
            doubleknd26 Kideok Kim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: