Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64843

Memory Leak : Hudson.model.hudson.labels (java.util.concurrent.ConcurrentHashMap)

      Since the beginning of the year,  we faced with Jenkins 2 master slowdowns : from GC logs, we suspect a memory leak ..

      As you can see, the Old Gen memory increased by more than 800 MB in two weeks.

      To confirm this memory leak suspicious  : we made a heap dump.

      The heap dump shows 51% of the old gen is used to store the attribute Hudson.model.hudson.labels which contains ConcurrentHashMap of 3517 labels used for each jobs launched from the starting of the jvm.

      The ConcurrentHashMap containing labels doesn't appear to remove entries after the job has run.

      This issue seems to be revealed since the workload on Jenkins has increased a lot in recent months.

       

       

          [JENKINS-64843] Memory Leak : Hudson.model.hudson.labels (java.util.concurrent.ConcurrentHashMap)

          500 Builds a day seems pretty normal usage, I don't see why there should be 3000 entries

          The random string will not be kept permanently (sorry I was not aware of how the operation of the podTemplates worked) as the random string is not tied to a podTemplate directly but to a specific node, once the node is deleted it should be removed.

          I'm curious what is actually in that particular Map (the actual values in the key / value field, especially the key)

          Raihaan Shouhell added a comment - 500 Builds a day seems pretty normal usage, I don't see why there should be 3000 entries The random string will not be kept permanently (sorry I was not aware of how the operation of the podTemplates worked) as the random string is not tied to a podTemplate directly but to a specific node, once the node is deleted it should be removed. I'm curious what is actually in that particular Map (the actual values in the key / value field, especially the key)

          Billy added a comment -

          As explain before, assuming the labels map should remove label node after the node deletion : the 3500 entries shows (by the heap dump) that the labels map doesn't have the expected behavior ..

          I don't found any fix about this issue in jenkins release notes > 2.249.2 : may be you have more informations on this.

          Could you confirm that this memory leak is caused by an issue in jenkins core or kubernetes plugin ? 

          Billy added a comment - As explain before, assuming the labels map should remove label node after the node deletion : the 3500 entries shows (by the heap dump) that the labels map doesn't have the expected behavior .. I don't found any fix about this issue in jenkins release notes > 2.249.2 : may be you have more informations on this. Could you confirm that this memory leak is caused by an issue in jenkins core or kubernetes plugin ? 

          Hey Billy,

          I took a look at the Kubernetes plugin and it seems that there was a bug with template deletion that is present in 1.27.3 https://github.com/jenkinsci/kubernetes-plugin/pull/880

          This might explain why the labels are not removed.

          Raihaan Shouhell added a comment - Hey Billy, I took a look at the Kubernetes plugin and it seems that there was a bug with template deletion that is present in 1.27.3 https://github.com/jenkinsci/kubernetes-plugin/pull/880 This might explain why the labels are not removed.

          htbthach I believe your issue should go away if you update the kubernetes plugin to 1.27.5 or perhaps just 1.29.0

          Raihaan Shouhell added a comment - htbthach I believe your issue should go away if you update the kubernetes plugin to 1.27.5 or perhaps just 1.29.0

          Billy added a comment -

          Thanks for your help Raihaan : i have looked the PR https://github.com/jenkinsci/kubernetes-plugin/pull/880 and this seems fix our issue.

          I missed this PR, the memory leaks weren't highlighted

          We will upgrade the kubernetes plugin to 1.27.5 : https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.27.5

          Hope this upgrade fix this issue.

          Billy added a comment - Thanks for your help Raihaan : i have looked the PR  https://github.com/jenkinsci/kubernetes-plugin/pull/880  and this seems fix our issue. I missed this PR, the memory leaks weren't highlighted We will upgrade the kubernetes plugin to 1.27.5 : https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.27.5 Hope this upgrade fix this issue.

          I'll leave this issue open, feel free to let me know if the upgrade solves the issue in case someone else runs into this issue the solution will be recorded here

          Raihaan Shouhell added a comment - I'll leave this issue open, feel free to let me know if the upgrade solves the issue in case someone else runs into this issue the solution will be recorded here

          htbthach Has your issue been resolved?

          Raihaan Shouhell added a comment - htbthach Has your issue been resolved?

          I'm going to resolve this issue. Feel free to reopen if this doesn't solve your issue

          Raihaan Shouhell added a comment - I'm going to resolve this issue. Feel free to reopen if this doesn't solve your issue

          Billy added a comment -

          We haven't updated Jenkins yet. After the update, i will notice you if this issue is fixed.
          Thank you for your help.

          Billy added a comment - We haven't updated Jenkins yet. After the update, i will notice you if this issue is fixed. Thank you for your help.

          Billy added a comment -

          Hi Raihann !

          Some weeks after the update, I can confirmed you that the upgrade of kubernetes plugin fix this memory leak issue.

          Thanks you for your time.

          Billy added a comment - Hi Raihann ! Some weeks after the update, I can confirmed you that the upgrade of kubernetes plugin fix this memory leak issue. Thanks you for your time.

            raihaan Raihaan Shouhell
            htbthach Billy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: