Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58528

Channel listeners (SlaveCommandStatistics.Statistics) retained after node removal

    XMLWordPrintable

Details

    • support-core-2.60

    Description

      OOM situation investigation revealed that com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:

      println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
      println "Computers " + Jenkins.instance.computers.size()
      

      Values from some of the instances with varying size/uptime:

      Listeneres 1990
      Computers 5
      ---
      Listeneres 62
      Computers 33
      ---
      Listeneres 16521
      Computers 16
      

       

      The instance torn down had 30K instances.

      Attachments

        Activity

          olivergondza Oliver Gondža created issue -
          olivergondza Oliver Gondža made changes -
          Field Original Value New Value
          Description OOM situation investigation revealed that \{com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:

           
          {noformat}
          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()

          {noformat}
           

          Values from some of the instances with varying size/uptime:
          {noformat}
          Listeneres 1990
          Computers 5

          ---

          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()

          ---

          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()

          {noformat}
           

          The instance torn down had 30K instances.
          OOM situation investigation revealed that \{com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:
          {noformat}
          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()
          {noformat}
          Values from some of the instances with varying size/uptime:
          {noformat}
          Listeneres 1990
          Computers 5
          ---
          Listeneres 62
          Computers 33
          ---
          Listeneres 16521
          Computers 16
          {noformat}
           

          The instance torn down had 30K instances.

          jglick, as an author of the report, would you say this makes sense preserving after computer is gone (vs. only while it is present)? I am considering when to rotate the records - at node removal or keeping it around for a while to ease investigation...

          olivergondza Oliver Gondža added a comment - jglick , as an author of the report, would you say this makes sense preserving after computer is gone (vs. only while it is present)? I am considering when to rotate the records - at node removal or keeping it around for a while to ease investigation...
          olivergondza Oliver Gondža made changes -
          Description OOM situation investigation revealed that \{com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:
          {noformat}
          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()
          {noformat}
          Values from some of the instances with varying size/uptime:
          {noformat}
          Listeneres 1990
          Computers 5
          ---
          Listeneres 62
          Computers 33
          ---
          Listeneres 16521
          Computers 16
          {noformat}
           

          The instance torn down had 30K instances.
          OOM situation investigation revealed that {{com.cloudbees.jenkins.support.impl.SlaveCommandStatistics#statistics}} is always growing. For instances with dynamically provisioned nodes this is effectively a memory leak. Detection:
          {noformat}
          println "Listeneres " + ExtensionList.lookupSingleton(com.cloudbees.jenkins.support.impl.SlaveCommandStatistics.class).@statistics.size()
          println "Computers " + Jenkins.instance.computers.size()
          {noformat}
          Values from some of the instances with varying size/uptime:
          {noformat}
          Listeneres 1990
          Computers 5
          ---
          Listeneres 62
          Computers 33
          ---
          Listeneres 16521
          Computers 16
          {noformat}
           

          The instance torn down had 30K instances.
          olivergondza Oliver Gondža added a comment - Fix proposed: https://github.com/jenkinsci/support-core-plugin/pull/180
          olivergondza Oliver Gondža made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jglick Jesse Glick added a comment -

          Yes we need to preserve statistics after agent removal. I commented in more detail in the PR.

          jglick Jesse Glick added a comment - Yes we need to preserve statistics after agent removal. I commented in more detail in the PR.
          egutierrez Evaristo Gutierrez made changes -
          Released As support-core-2.60
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          aheritier Arnaud Héritier made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            olivergondza Oliver Gondža
            olivergondza Oliver Gondža
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: