Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56376

remoting kakfa plugin creates too many threads and takes too much cpu time

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Progress (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: remoting-kafka-plugin
    • Labels:
      None
    • Environment:
      Jenkins-2.165
      remoting-kafka-1.1.3
    • Similar Issues:

      Description

      Only discovered when I moved to the production environment that the server and client side of this plugin are taking too much user CPU time. I think it is because the large number of threads (hopefully unnecessary) it spawns. Also, it might be consuming unnecessary memory.

      In the agent side, even when there are no jobs assigned it consumes a full cpu, and create about 30-40 threads. A naive thought is telling me that we should open as many threads as the number of executors the agent is configured with.

      On the server side, when I have about 8-10 kafka agent nodes it takes ~7 cpus (of my 8 cores), with kafka Channel reader threads that are opened for the agents.
      In addition to the 54 threads that are created for the kafka-producer-network thread
      In my test Jenkins instance, I observed the creation of 900+ threads

      I tested with the inside the docker container that is provided with the plugin and also observed it creates a high number of threads.

      I'll upload some screenshots.

        Attachments

          Activity

          Hide
          fnaum Federico Naum added a comment -

          Won't be able to upload the screenshots until Wed. (some issue with the firewall) I 'll see If I can get around it.
          But just installing the monitoring plugin and then ordering by cpu time you'll those Channel reader threads chewing all the juice

          Show
          fnaum Federico Naum added a comment - Won't be able to upload the screenshots until Wed. (some issue with the firewall) I 'll see If I can get around it. But just installing the monitoring plugin and then ordering by cpu time you'll those Channel reader threads chewing all the juice
          Hide
          fnaum Federico Naum added a comment -

          Hi Pham Vu Tuan was there any progress with this? or should be moved to On Hold?

          Show
          fnaum Federico Naum added a comment - Hi Pham Vu Tuan was there any progress with this? or should be moved to On Hold?
          Hide
          pvtuan10 Pham Vu Tuan added a comment -

          Hi Federico Naum, sorry for the delay, I haven't found a suitable time slot to work on this yet, this can be considered "On Hold" at this moment. Sorry for any convinience caused, I will inform you as soon as this get fixed.

          Show
          pvtuan10 Pham Vu Tuan added a comment - Hi Federico Naum , sorry for the delay, I haven't found a suitable time slot to work on this yet, this can be considered "On Hold" at this moment. Sorry for any convinience caused, I will inform you as soon as this get fixed.
          Hide
          fnaum Federico Naum added a comment -

          Thank you, I appreciate your efforts.

          Show
          fnaum Federico Naum added a comment - Thank you, I appreciate your efforts.
          Hide
          mayur_kumar Mayur Kumar added a comment -

          Hi Pham Vu Tuan,

           

          We have a disconnection issue over TCP and were looking at this plugin as a potential solution to the issue. Are there any plans on for fixing this issue, or is there anyone currently working on it?

          Show
          mayur_kumar Mayur Kumar added a comment - Hi Pham Vu Tuan ,   We have a disconnection issue over TCP and were looking at this plugin as a potential solution to the issue. Are there any plans on for fixing this issue, or is there anyone currently working on it?

            People

            Assignee:
            pvtuan10 Pham Vu Tuan
            Reporter:
            fnaum Federico Naum
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: