Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72564

OOM periodically with a growing size of config.xml

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • gradle-plugin, kubernetes-plugin

      Description

      We've suffered from a periodically crush of jenkins in past a few months. After some investigation, we found out that the jenkins config.xml file size has been increasing periodically and finally lead to a OOME.

      What happened

      There are tons of hudson.slaves.EnvironmentVariablesNodeProperty declaring MAVEN_OPTS in our config.xml file,  and there's a screenshot showing how big this config.xml file is (we restart jenkins and reset this file periodically, and the first time we found out this problem, this file is up to 300mb big).

      What did we try

      We made config.xml file read-only by `chattr +i config.xml`, and found out that gradle plugin is trying to add MAVEN_OPTS every-time we start a pipeline job.

      What is weird

      Our jenkins is working with a kubernetes backend, and it seems like that it only happens when new pods are delivered by built-in node, but we've configured number of executors to 0 for built-in node.

      What i want to know

      1. why built-in node still taking job builds when its "number of executors" is zero?

      2. we've disabled gradle plugin to fix this problem, but what should we do if we need this plugin?

        1. image-2024-01-16-12-52-02-459.png
          image-2024-01-16-12-52-02-459.png
          487 kB
        2. image-2024-01-16-12-57-55-968.png
          image-2024-01-16-12-57-55-968.png
          14 kB
        3. image-2024-01-16-13-04-39-493.png
          image-2024-01-16-13-04-39-493.png
          48 kB

          [JENKINS-72564] OOM periodically with a growing size of config.xml

          Iurii Ignatko added a comment - - edited

          Hi wangchen!
          I think the problem you're referring to has been fixed in this ticket - https://issues.jenkins.io/browse/JENKINS-70663.

          Based off of the ticket description, you're using Gradle Plugin 1.39.1 which is quite old. I would recommend you consider upgrading to latter versions, as they contain multiple improvements (this one in particular).

          In your case, version 2.3.2 should be enough, but feel free to check ones higher - https://github.com/jenkinsci/gradle-plugin/releases.

           

          Iurii Ignatko added a comment - - edited Hi wangchen ! I think the problem you're referring to has been fixed in this ticket - https://issues.jenkins.io/browse/JENKINS-70663 . Based off of the ticket description, you're using Gradle Plugin 1.39.1 which is quite old. I would recommend you consider upgrading to latter versions, as they contain multiple improvements (this one in particular). In your case, version 2.3.2 should be enough, but feel free to check ones higher - https://github.com/jenkinsci/gradle-plugin/releases .  

          wang added a comment -

          Thanks welandaz! 

          After an upgrade of gradle plugin (to 2.3.2) in our staging environment, the config.xml file size stopped to grow with empty and worthless MAVEN_OPTS variable, thus will not occupy tons of memory during runtime.

          But built-in node still taking our pipeline jobs for a short duration, normally a couple of hundred milliseconds and these builds are then forwarded to Pods created by Kubernetes. I've searched over jenkins community and stackoverflow, but there's no clear answer to it.

          Like this from community: https://community.jenkins.io/t/why-are-builds-appearing-on-built-in-node-but-its-not-using-the-executors-assigned-causing-high-memory-issue/9666, explained that parts of our pipeline runs on the controller node regardless of jenkins file

          wang added a comment - Thanks welandaz !   After an upgrade of gradle plugin (to 2.3.2) in our staging environment, the config.xml file size stopped to grow with empty and worthless MAVEN_OPTS variable, thus will not occupy tons of memory during runtime. But built-in node still taking our pipeline jobs for a short duration, normally a couple of hundred milliseconds and these builds are then forwarded to Pods created by Kubernetes. I've searched over jenkins community and stackoverflow, but there's no clear answer to it. Like this from community: https://community.jenkins.io/t/why-are-builds-appearing-on-built-in-node-but-its-not-using-the-executors-assigned-causing-high-memory-issue/9666, explained that parts of our pipeline runs on the controller node regardless of jenkins file

          Iurii Ignatko added a comment -

          hi wangchen !
          I'm glad to hear the upgrade fixed the issue!

          Regarding another problem you've mentioned - it doesn't seem to me that it's a Gradle Plugin problem per se (but, please, correct me if I'm wrong). It sounds like a different issue related to scheduling in pipeline jobs (Jenkins specific, not plugin specific).

          I'd suggest you open another ticket specifically for that and we close this one as the upgrade of the plugin fixed the issue for you.

          Iurii Ignatko added a comment - hi wangchen ! I'm glad to hear the upgrade fixed the issue! Regarding another problem you've mentioned - it doesn't seem to me that it's a Gradle Plugin problem per se (but, please, correct me if I'm wrong). It sounds like a different issue related to scheduling in pipeline jobs (Jenkins specific, not plugin specific). I'd suggest you open another ticket specifically for that and we close this one as the upgrade of the plugin fixed the issue for you.

          wang added a comment - - edited

          welandaz okay, thank you for help

          wang added a comment - - edited welandaz okay, thank you for help

            welandaz Iurii Ignatko
            wangchen wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: