Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56376

remoting kakfa plugin creates too many threads and takes too much cpu time

    XMLWordPrintable

Details

    • Bug
    • Status: In Progress (View Workflow)
    • Major
    • Resolution: Unresolved
    • remoting-kafka-plugin
    • None
    • Jenkins-2.165
      remoting-kafka-1.1.3

    Description

      Only discovered when I moved to the production environment that the server and client side of this plugin are taking too much user CPU time. I think it is because the large number of threads (hopefully unnecessary) it spawns. Also, it might be consuming unnecessary memory.

      In the agent side, even when there are no jobs assigned it consumes a full cpu, and create about 30-40 threads. A naive thought is telling me that we should open as many threads as the number of executors the agent is configured with.

      On the server side, when I have about 8-10 kafka agent nodes it takes ~7 cpus (of my 8 cores), with kafka Channel reader threads that are opened for the agents.
      In addition to the 54 threads that are created for the kafka-producer-network thread
      In my test Jenkins instance, I observed the creation of 900+ threads

      I tested with the inside the docker container that is provided with the plugin and also observed it creates a high number of threads.

      I'll upload some screenshots.

      Attachments

        Activity

          fnaum Federico Naum added a comment -

          Won't be able to upload the screenshots until Wed. (some issue with the firewall) I 'll see If I can get around it.
          But just installing the monitoring plugin and then ordering by cpu time you'll those Channel reader threads chewing all the juice

          fnaum Federico Naum added a comment - Won't be able to upload the screenshots until Wed. (some issue with the firewall) I 'll see If I can get around it. But just installing the monitoring plugin and then ordering by cpu time you'll those Channel reader threads chewing all the juice
          fnaum Federico Naum added a comment -

          Hi pvtuan10 was there any progress with this? or should be moved to On Hold?

          fnaum Federico Naum added a comment - Hi pvtuan10 was there any progress with this? or should be moved to On Hold?
          pvtuan10 Pham Vu Tuan added a comment -

          Hi fnaum, sorry for the delay, I haven't found a suitable time slot to work on this yet, this can be considered "On Hold" at this moment. Sorry for any convinience caused, I will inform you as soon as this get fixed.

          pvtuan10 Pham Vu Tuan added a comment - Hi fnaum , sorry for the delay, I haven't found a suitable time slot to work on this yet, this can be considered "On Hold" at this moment. Sorry for any convinience caused, I will inform you as soon as this get fixed.
          fnaum Federico Naum added a comment -

          Thank you, I appreciate your efforts.

          fnaum Federico Naum added a comment - Thank you, I appreciate your efforts.
          mayur_kumar Mayur Kumar added a comment -

          Hi pvtuan10,

           

          We have a disconnection issue over TCP and were looking at this plugin as a potential solution to the issue. Are there any plans on for fixing this issue, or is there anyone currently working on it?

          mayur_kumar Mayur Kumar added a comment - Hi pvtuan10 ,   We have a disconnection issue over TCP and were looking at this plugin as a potential solution to the issue. Are there any plans on for fixing this issue, or is there anyone currently working on it?

          People

            pvtuan10 Pham Vu Tuan
            fnaum Federico Naum
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: